{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyMDzW0YlJwJXYt8vvLBJn5C",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LC1332/Chat-Haruhi-Suzumiya/blob/main/notebook/test_yuki_builder.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Chat凉宫春日 Chat-Haruhi-Suzumiya\n",
        "\n",
        "[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)]()\n",
        "[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)]()\n",
        "[![Huggingface Gradio](https://img.shields.io/static/v1?label=Demo&message=Huggingface%20Gradio&color=orange)](https://huggingface.co/spaces/silk-road/ChatHaruhi)\n",
        "\n",
        "\n",
        "本脚本是李鲁鲁开发的YukiBuilder的测试工具，后者由\n",
        "\n",
        "李鲁鲁，封小洋，scixing，aria Fei等共同开发\n",
        "\n",
        "---\n",
        "\n",
        "**Chat凉宫春日**是模仿凉宫春日等一系列动漫人物，使用近似语气、个性和剧情聊天的语言模型，\n",
        "\n",
        "<details>\n",
        "  <summary> 本项目由李鲁鲁，冷子昂，闫晨曦，封小洋，scixing，沈骏一，Aria Fei, 米唯实, 吴平宇, 贾曜恺等开发。 </summary>\n",
        "\n",
        "李鲁鲁发起了项目，并完成了最早的版本，在多个微信群实现了测试。完成了训练数据自动生成对话部分，设计了整体的路线，并撰写报告。\n",
        "\n",
        "冷子昂负责了每一个阶段的Gradio开发，以及每个部分的功能整合和架构设计。\n",
        "\n",
        "闫晨曦一开始将李鲁鲁的notebook重构为app.py，参与了WebUI的embedding部分重构整合。\n",
        "\n",
        "封小洋进行了中文转日文模型的选型，完成了针对台词抽取图片的工具。整合了声纹识别。即将继续参加台本工具的开发。\n",
        "\n",
        "scixing实践了VITS语音，完成了台词对应的语音抽取，实现了第一个版本的声纹分类。\n",
        "\n",
        "沈骏一实现了使用ChatGLM2 finetune实验\n",
        "\n",
        "Aria(Yaying Fei)对接了whisper到台本工具。即将继续参加台本工具的开发。\n",
        "\n",
        "米唯实实现了Chat哆啦A梦的分支版本\n",
        "\n",
        "吴平宇部署了ChatGLM2的训练程序，并提供了训练的计算资源。\n",
        "\n",
        "贾曜恺开发了一个Vue前端实现方案\n",
        "\n",
        "</details>"
      ],
      "metadata": {
        "id": "MazvvpHTxXY6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 部署"
      ],
      "metadata": {
        "id": "VDK-NK1lxuwk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# pip install if necessary"
      ],
      "metadata": {
        "id": "w5L0W74gxwu7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/LC1332/Chat-Haruhi-Suzumiya"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o7Lx7iPsxyg7",
        "outputId": "5fcbad11-cd48-42c2-acb4-49d251b460ef"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'Chat-Haruhi-Suzumiya'...\n",
            "remote: Enumerating objects: 3689, done.\u001b[K\n",
            "remote: Counting objects: 100% (1677/1677), done.\u001b[K\n",
            "remote: Compressing objects: 100% (1352/1352), done.\u001b[K\n",
            "remote: Total 3689 (delta 397), reused 1553 (delta 312), pack-reused 2012\u001b[K\n",
            "Receiving objects: 100% (3689/3689), 91.56 MiB | 26.20 MiB/s, done.\n",
            "Resolving deltas: 100% (1297/1297), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/Chat-Haruhi-Suzumiya/yuki_builder/\n",
        "!ls"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qUHaPJYiz1Ol",
        "outputId": "b98172fb-9f0d-47cc-a533-5d26e358eb66"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/Chat-Haruhi-Suzumiya/yuki_builder\n",
            "audio_feature_ext  input_folder  requirements_run_whisper.txt  srt2csv.py\n",
            "crop.py\t\t   recognize.py  run_whisper.py\t\t       yuki_builder.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# install requirement from requirements_run_whisper.txt\n",
        "!pip install -q -r requirements.txt\n"
      ],
      "metadata": {
        "id": "1-7K1UxwDArc",
        "outputId": "eede75bd-3f66-4200-9910-9a7874dd0044",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/792.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m358.4/792.9 kB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m792.9/792.9 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m60.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for openai-whisper (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for ffmpeg (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# mount google drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PhMtbZEQhJcC",
        "outputId": "d932a253-c97f-4906-96ba-19580c7b5d84"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "对于我来说测试文件都在 /content/drive/Mydrive/GPTData/YukiTestData 中间\n",
        "\n",
        "whisper测试文件\n",
        "\n",
        "https://drive.google.com/file/d/1ZwexhJoeHqUAHosEJQuas9WaFVQCKVLq/view?usp=sharing\n",
        "\n",
        "ass文件\n",
        "\n",
        "https://drive.google.com/file/d/1QzP3Y2fqSAKU4tEYrRMwFsPXw1TlGgFs/view?usp=sharing\n",
        "\n",
        "\n",
        "mkv文件\n",
        "\n",
        "https://drive.google.com/file/d/1TXCBOOk-sw95_PJ9nLAg-P4JDV9-twWa/view?usp=sharing\n",
        "\n",
        "annotate文件\n",
        "https://drive.google.com/file/d/12eb3YHed_9TAUym6L0nzpWaMZ4Wd-Moj/view?usp=sharing\n",
        "\n",
        "annotate_map文件\n",
        "\n",
        "https://drive.google.com/file/d/1GrBcG-7g-QW4KlmV1GbHz12y4d7QXZgL/view?usp=sharing"
      ],
      "metadata": {
        "id": "0X8hK10shZqT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls /content/drive/MyDrive/GPTData/YukiTestData"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9c1y9bwahk_r",
        "outputId": "b078894a-f8b9-4387-cdda-aea17b357cf9"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][BDRIP][1920x1080][x264_FLAC_2][CE13579E].mkv'\n",
            "'[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][x264_Vorbis][DVDRip].ass'\n",
            " Haruhi_03_annotate.csv\n",
            " Haruhi_03_annotate.txt\n",
            " haruhi_EP3_annotate_map.csv\n",
            " whisperTest.mp4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q ass pysrt pysubs2"
      ],
      "metadata": {
        "id": "Os8i-fK1eONP",
        "outputId": "058bb6db-4b82-426c-8a27-f5c206d636fe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/104.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.4/104.4 kB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for pysrt (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cp -r /content/drive/MyDrive/GPTData/YukiTestData/. /content/input/"
      ],
      "metadata": {
        "id": "ETg_a50Ejduy"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 测试whisper"
      ],
      "metadata": {
        "id": "SGxpGJ-dxqZZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "使用whisperTest.mp4进行测试"
      ],
      "metadata": {
        "id": "teIWOnweh0jU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i8xcgnyfxL8q",
        "outputId": "7feb1ea4-c6f9-457a-a652-fb2eb2bff05f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "warning srt_folder is not exist\n",
            "create folder /content/srt\n",
            "---loading model in your local path or downloading now---\n",
            " 76%|████████████████████████████▉         | 1.08G/1.42G [00:10<00:03, 109MiB/s]\n",
            "Traceback (most recent call last):\n",
            "  File \"/content/Chat-Haruhi-Suzumiya/yuki_builder/yuki_builder.py\", line 36, in <module>\n",
            "    run_whisper(args)\n",
            "  File \"/content/Chat-Haruhi-Suzumiya/yuki_builder/run_whisper.py\", line 102, in run_whisper\n",
            "    result = Video2Subtitles().transcribe(input_file, srt_folder)\n",
            "  File \"/content/Chat-Haruhi-Suzumiya/yuki_builder/run_whisper.py\", line 28, in __init__\n",
            "    self.model = whisper.load_model(MODEL_WHISPER)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/whisper/__init__.py\", line 131, in load_model\n",
            "    checkpoint_file = _download(_MODELS[name], download_root, in_memory)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/whisper/__init__.py\", line 85, in _download\n",
            "    raise RuntimeError(\n",
            "RuntimeError: Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model.\n"
          ]
        }
      ],
      "source": [
        "!python yuki_builder.py whisper \\\n",
        "                       -input_video /content/input/whisperTest.mp4 \\\n",
        "                       -srt_folder /content/srt"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 测试srt2csv"
      ],
      "metadata": {
        "id": "6d5sO7Nz0IIj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python yuki_builder.py srt2csv \\\n",
        "                      -input_srt \"/content/input/[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][x264_Vorbis][DVDRip].ass\" \\\n",
        "                      -srt_folder /content/srt"
      ],
      "metadata": {
        "id": "rbvKFD3A0D_C"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python yuki_builder.py srt2csv \\\n",
        "                      -input_srt \"/content/srt/whisperTest.srt\" \\\n",
        "                      -srt_folder /content/srt/"
      ],
      "metadata": {
        "id": "Vu4i4kai4bfB",
        "outputId": "a6198691-9b2e-4b5e-8683-08c1b6071896",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Error: The input file /content/srt/whisperTest.srt is not exist\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 测试人物切分"
      ],
      "metadata": {
        "id": "_T8hU3hi0ctK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "三个文件分别在\n",
        "\n",
        "百度链接：\n",
        "\n",
        "链接: https://pan.baidu.com/s/1Z-d4mqEbE8885HKqVqxIlQ?pwd=kmvx 提取码: kmvx\n",
        "\n",
        "\n",
        "谷歌连接：\n",
        "\n",
        "https://drive.google.com/file/d/12eb3YHed_9TAUym6L0nzpWaMZ4Wd-Moj/view?usp=sharing,\n",
        "\n",
        "https://drive.google.com/file/d/1GrBcG-7g-QW4KlmV1GbHz12y4d7QXZgL/view?usp=sharing,\n",
        "\n",
        "https://drive.google.com/file/d/1TXCBOOk-sw95_PJ9nLAg-P4JDV9-twWa/view?usp=sharing"
      ],
      "metadata": {
        "id": "yABhy-H6--4y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python yuki_builder.py crop \\\n",
        "                      -annotate_map /content/input/haruhi_EP3_annotate_map.csv \\\n",
        "                      -role_audios /content/role_audios"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cCzM9kZC0YYp",
        "outputId": "0670eeb8-7ebd-4686-a265-3de9382ba4b4"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "role_audios is not exist\n",
            "video clip by csv file start: 100% 440/440 [00:33<00:00, 13.29it/s]\n",
            "downloading model pth model.pth\n",
            "model.pth success download\n",
            "downloading model pth model.state\n",
            "model.state success download\n",
            "downloading model pth optimizer.pth\n",
            "optimizer.pth success download\n",
            "成功加载模型参数和优化方法参数：./audio_feature_ext/models/model.pth\n",
            "extract 长门 audio features ,convert .wav to .pkl: 100% 60/60 [00:16<00:00,  3.68it/s]\n",
            "extract 朝比奈 audio features ,convert .wav to .pkl: 100% 65/65 [00:01<00:00, 34.66it/s]\n",
            "extract 阿虚 audio features ,convert .wav to .pkl: 100% 163/163 [00:05<00:00, 29.54it/s]\n",
            "extract 古泉 audio features ,convert .wav to .pkl: 100% 76/76 [00:01<00:00, 41.21it/s]\n",
            "extract 春日 audio features ,convert .wav to .pkl: 100% 76/76 [00:01<00:00, 45.95it/s]\n",
            "音频特征提取完成\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 测试识别"
      ],
      "metadata": {
        "id": "FesuIVOX0lEv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python yuki_builder.py recognize \\\n",
        "                      -input_video \"/content/input/[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][BDRIP][1920x1080][x264_FLAC_2][CE13579E].mkv\" \\\n",
        "                      -input_srt \"/content/input/[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][x264_Vorbis][DVDRip].ass\" \\\n",
        "                      -role_audios /content/role_audios \\\n",
        "                      -output_folder /content/final_output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MmJbRoYK0i0R",
        "outputId": "e6c3122c-338f-4fa6-f960-83dd62fd6014"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "warning output_folder is not exist\n",
            "create folder /content/final_output\n",
            "成功加载模型参数和优化方法参数：./audio_feature_ext/models/model.pth\n",
            "extract [CASO][Suzumiya_Haruhi_no_Yuuutsu][03][BDRIP][1920x1080][x264_FLAC_2][CE13579E] audio features ,convert .wav to .pkl: 100% 452/452 [00:14<00:00, 30.39it/s]\n",
            "音频特征提取完成\n",
            "识别结果保存到csv, /content/final_output/[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][BDRIP][1920x1080][x264_FLAC_2][CE13579E]_output.csv\n",
            "识别结果保存到txt, /content/final_output/[CASO][Suzumiya_Haruhi_no_Yuuutsu][03][BDRIP][1920x1080][x264_FLAC_2][CE13579E]_output.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "zN3DQTvP0q1l"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}
